Three-dimensional computational imaging method and apparatus based on single-pixel sensor, and non-transitory computer-readable storage medium

ABSTRACT

The present disclosure proposes a three-dimensional computational imaging method and apparatus based on a single-pixel sensor, and a storage medium. The method includes the following. A stripe coding is combined with a two-dimensional imaging coding through a preset optical coding to generate a new optical coding, and the new optical coding is loaded into an spatial light modulator (SLM); a two-dimensional spatial information and depth information of a scene are coupled into a one-dimensional measurement value by using a single-pixel detector and the SLM loaded with the new optical coding; and the two-dimensional spatial information and the depth information of the scene are reconstructed, from the one-dimensional measurement value through a decoupling algorithm, for three-dimensional imaging.

CROSS-REFERENCE TO RELATED APPLICATION

The present application claims priority to Chinese Patent ApplicationNo. 201910645486.7, filed Jul. 17, 2019, the entire disclosure of whichis incorporated herein by reference.

FIELD

The present disclosure relates to a field of a computational photographytechnology, particularly, to a three-dimensional computational imagingmethod and apparatus based on a single-pixel sensor, and anon-transitory computer-readable storage medium.

BACKGROUND

The traditional camera uses a two-dimensional sensor unit array as aphotosensitive element, and a sensor may only sense intensitysuperposition information of light, so that depth information of a sceneis lost in an imaging process. Therefore, a traditional photographytechnology acquires a projection of a three-dimensional world on atwo-dimensional plane, and an obtained single image cannot accuratelyrecover the depth and structure information of a three-dimensionalscene, therefore it cannot bring a stereoscopic impression to humaneyes.

A traditional three-dimensional measurement technique includes thefollowings. A binocular intersection measuring method which utilizes aplurality of cameras to perform a multi-angle shooting and is combinedwith a feature point detection; a laser scanning method which recordsthree-dimensional coordinates of surface dense points of a measuredobject by using a principle of laser ranging and reconstructsthree-dimensional model data of the measured object; an opticalinterferometry which obtains a phase by observing interference fringesgenerated by an interferometer, and the like. The above methodsrespectively need to use apparatuses such as a multi-camera, a laserscanning, a grating, a lens and the like in an experimental process,which has disadvantages of a complex optical path, a heavy system and ahigher cost.

A three-dimensional imaging study in computer vision typically utilizemultiple images or a 3D scanning device to recover a depth and astructure of a scene. Following methods are used in the related arts:(1) a three-dimensional curved surface measurement technology applied ina reverse engineering, which adopts a multi-view image to recover thedepth; (2) a three-dimensional imaging data processing method of aflight time method, which continuously sends light pulses to a target,receives the light returned from the object by using a sensor, anddetects a round-trip time of the light pulses to obtain distances(heights) of different space points of the target; (3) a structuredlight three-dimensional imaging technology research, which provides athree-dimensional imaging method based on Fourier fringe analysis,projects carrier frequency fringes to an object surface by using agrating device, and then obtains a deformed two-dimensional fringe imagemodulated by the structured light fringes on the object surface from aspecific angle. The above methods usually require a multi-view camera oran additional active light source device, which increases a complexityof the hardware system.

In recent years, an emerging computational photography combines acomputing, a digital sensor, an optical system, an intelligentillumination and the like, and improves a traditional camera on anaspect of an imaging mechanism so as to acquire more visual information.For example, in a single-camera light field acquisition method based ona light field theory, angle information lost by a traditional camera isacquired in a single exposure by sacrificing a spatial resolution; in akinect depth camera published by Microsoft Corp. in 2010, a projectorcalibration is converted into a mature camera calibration byestablishing a corresponding relation between a projector image and acamera image, so that a single-camera structured light three-dimensionalmeasurement system is converted into a classical binocular vision systemfor a depth measurement.

However, a current acquisition mode of a single camera cannoteffectively acquire the depth information of the scene through a smallnumber of test times. The acquisition system based on the light fieldtheory essentially only acquires angular information of the scene butnot the depth information of the scene, and the spatial resolution issacrificed at the same time when acquiring the angular information.Therefore, there is a need to search for a method for efficiently andstably performing depth information acquisition, thereby performing anaccurate depth estimation.

A single-pixel camera as a new imaging device is characterized by havinga sensing unit, and a signal-to-noise ratio of the single-pixel camerais higher because the single-pixel camera only needs a photosensitivedetector. In addition, the single-pixel detector has a wider spectralresponse range. Moreover, the single-pixel detector can reducerequirements of data acquisition, transmission and storage based on acompressed sensing theory, so that the single-pixel detector hasincomparable advantages compared with a traditional array sensor and hasa wide application range.

In recent decades, the single-pixel detector has been used for atwo-dimensional imaging of a wide-spectrum, such as a multi-spectralimaging, an infrared imaging, a terahertz imaging, an X-ray diffractiontomography, and the like, but an exploration of three-dimensionalimaging is still in the beginning. At present, a research proposes thata plurality of single-pixel detectors are placed at different spatialpositions, so as to acquire and reconstruct scene images with differentvisual angles and finally synthesize three-dimensional information ofthe scene; in addition, with respect to a three-dimensional imagingsystem with an improved time-of-flight based on a single-pixel, thearticle records a complete time of a scattered light reaching a detectorby using a single-pixel detector, so that an acquisition time and areconstruction time of a time-of-flight method are reduced; a currentfirst single-pixel three-dimensional camera utilizes a quantum imagingtechnology and depends on detection and calculation of reflected lightwave energy to obtain a depth of field; a single-pixel three-dimensionalimaging method based on an LED array utilizes a combination of colorimages generated by photodiodes at different positions to obtain athree-dimensional structure. However, the above methods have highcomplexity and high cost, so that a method of realizing an efficientthree-dimensional imaging by using a single-pixel sensor has importantsignificance.

SUMMARY

The present disclosure seeks to solve at least one of the problemsexisting in the related art to at least some extent.

Accordingly, an objective of the present disclosure is to provide athree-dimensional computational imaging method based on a single-pixelsensor.

Another objective of the present disclosure is to provide athree-dimensional computational imaging apparatus based on asingle-pixel sensor.

Another objective of the present disclosure is to provide anon-transitory computer-readable storage medium.

In order to achieve the above objectives, embodiments of the presentdisclosure provide the three-dimensional computational imaging methodbased on the single-pixel sensor. The method includes followings. Astripe coding is combined with a two-dimensional imaging coding througha preset optical coding, a new optical coding is generated, and the newoptical coding is loaded into an optical modulator; a two-dimensionalspatial information and depth information of a scene are coupled into aone-dimensional measurement value by using a single-pixel detector andthe optical modulator loaded with the new optical coding; and thetwo-dimensional spatial information and the depth information of thescene are reconstructed, from the one-dimensional measurement valuethrough a decoupling algorithm, for three-dimensional imaging.

In order to achieve the above objectives, embodiments of the presentdisclosure provide a three-dimensional computational imaging apparatusbased on a single-pixel sensor. The apparatus includes one or moreprocessors; a memory storing instructions executable by the one or moreprocessors; in which the one or more processors are configured to:combine a stripe coding with a two-dimensional imaging coding through apreset optical coding, generate a new optical coding, and load the newoptical coding into an optical modulator; couple a two-dimensionalspatial information and depth information of a scene into aone-dimensional measurement value by using a single-pixel detector andthe optical modulator loaded with the new optical coding; reconstruct,from the one-dimensional measurement value through a decouplingalgorithm, the two-dimensional spatial information and the depthinformation of the scene for three-dimensional imaging.

Embodiments of the present disclosure provide a non-transitorycomputer-readable storage medium having stored therein instructionsthat, when executed by a processor of a device, cause the processor toperform a three-dimensional computational imaging method based on asingle-pixel sensor, and the method includes: combining a stripe codingwith a two-dimensional imaging coding through a preset optical coding,generating a new optical coding, and loading the new optical coding intoan optical modulator; coupling a two-dimensional spatial information anddepth information of a scene into a one-dimensional measurement value byusing a single-pixel detector and the optical modulator loaded with thenew optical coding; and reconstructing, from the one-dimensionalmeasurement value through a decoupling algorithm, the two-dimensionalspatial information and the depth information of the scene forthree-dimensional imaging.

Additional aspects and advantages of embodiments of the presentdisclosure will be given in part in the following descriptions, becomeapparent in part from the following descriptions, or be learned from thepractice of the embodiments of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or additional aspects and advantages of embodiments of thepresent disclosure will become apparent and more readily appreciatedfrom the following descriptions made with reference to the drawings, inwhich:

FIG. 1 is a flow chart of a three-dimensional computational imagingmethod based on a single-pixel sensor according to an embodiment of thepresent disclosure;

FIG. 2 is a schematic diagram of an optical path of a conventionalsingle-pixel imaging system according to an embodiment of the presentdisclosure;

FIG. 3 is a schematic diagram of a system for a three-dimensionalcomputational imaging method based on a single-pixel sensor according toan embodiment of the present disclosure;

FIG. 4 is a schematic diagram of an optical path of a projected gratingphase method according to an embodiment of the present disclosure;

FIG. 5 is a schematic diagram of an integrity restoration of a softwarealgorithm according to an embodiment of the present disclosure; and

FIG. 6 is a structural diagram of a three-dimensional computationalimaging apparatus based on a single-pixel sensor according to anembodiment of the present disclosure.

DETAILED DESCRIPTION

Embodiments of the present disclosure will be described in detail andexamples of embodiments are illustrated in the drawings. The same orsimilar elements and the elements having the same or similar functionsare denoted by like reference numerals throughout the descriptions.Embodiments described herein with reference to drawings are explanatory,serve to explain the present disclosure, and are not construed to limitembodiments of the present disclosure. A three-dimensional computationalimaging method and apparatus based on a single-pixel sensor according toembodiments of the present disclosure will be described with referenceto the drawings. First, the three-dimensional computational imagingmethod based on the single-pixel sensor according to embodiments of thepresent disclosure will be described with reference to the drawings.

FIG. 1 is a flow chart of the three-dimensional computational imagingmethod based on the single-pixel sensor according to an embodiment ofthe present disclosure.

As illustrated in FIG. 1, the three-dimensional computational imagingmethod based on the single-pixel sensor includes the followings.

At block S101, a stripe coding is combined with a two-dimensionalimaging coding through a preset optical coding, a new optical coding isgenerated, and the new optical coding is loaded into an opticalmodulator.

It will be appreciated that, in the embodiment of the presentdisclosure, a new optical coding mode is adopted, by combining thestripe coding with the two-dimensional imaging coding, the new opticalcoding is generated and loaded into the optical modulator (the so calledlight modulator, such as a spatial light modulator (SLM)). In oneembodiment of the present disclosure, the two-dimensional imaging codingincludes, but is not limited to, one or more of a random coding, aHadamard coding, and a stripe coding.

Further, in one embodiment of the present disclosure, combining thestripe coding with the two-dimensional imaging coding through the presetoptical coding includes the followings. A matrix of a stripe codingmultiplied by a matrix of the two-dimensional imaging coding is presetto obtain a new matrix as an input of the optical modulator, so as toeffectively encode the two-dimensional spatial information and the depthinformation of the scene simultaneously.

It will be appreciated that, in embodiments of the present disclosure,the matrix of the stripe coding multiplied by the matrix of thetwo-dimensional imaging coding is preset to obtain the new matrix as theinput of a structured optical modulator, so as to effectively encode thetwo-dimensional spatial information and the depth information of thescene simultaneously.

At block S102, a two-dimensional spatial information and depthinformation of a scene are coupled into a one-dimensional measurementvalue by using a single-pixel detector and the optical modulator loadedwith the new optical coding.

It will be appreciated that, the modulated two-dimensional spatialinformation and the depth information of the scene are coupled into theone-dimensional measurement using a single-pixel detector. Thesingle-pixel detector and the spatial light modulator are used forcompleting a coding coupling acquisition of the three-dimensionalinformation of the scene.

At block S103, the two-dimensional spatial information and the depthinformation of the scene for three-dimensional imaging are reconstructedfrom the one-dimensional measurement value through a decouplingalgorithm.

It will be appreciated that, in the embodiment of the presentdisclosure, the two-dimensional information and the depth information ofthe scene may be reconstructed with a high precision by using thedecoupling algorithm. That is, in embodiments of the present disclosure,the decoupling algorithm based on a compressed sensing method, analternative projection method or a deep learning theory to reconstructthe two-dimensional spatial information of the scene with the highprecision from the one-dimensional measurement value. The decouplingalgorithm includes one or more of a non-iteration (a matrix inversion, atraditional correlation, a differential ghost Imaging (DGI)) method, alinear iteration (a gradient descent (GD), a conjugate gradient descent(CGD) method, a Poisson maximum likelihood (PSO) method, an alternativeprojection (AP)) method, a nonlinear iteration (a sparse representationcompressed sensing method (SR-CS), a total variation compressive sensingmethod (TV-CS)) and a deep learning method, so as to perform atwo-dimensional spatial information decoupling.

Further, in one embodiment of the present disclosure, reconstructing thetwo-dimensional spatial information and the depth information of thescene for three-dimensional imaging includes the followings. Thereconstruction is performed through the decoupling algorithm to obtainthe deformed optical field image.

It will be appreciated that, in embodiments of the present disclosure,the deformed optical field image is obtained by using a two-dimensionalspatial decoupling method, then a phase distribution is calculatedaccording to a phase shift function according to a discrete phase shifttechnology to obtain a scene depth.

Further, in one embodiment of the present disclosure, reconstructing thetwo-dimensional spatial information and the depth information of thescene for three-dimensional imaging further includes the following. Thephase distribution is calculated through a phase measurement profilemethod; and height information of an object surface is obtained by usinga geometric relation according to the phase distribution.

It will be appreciated that, a PMP (phase measurement profilometry) maybe adopted in embodiments of the present disclosure, the phasedistribution is calculated by an N-step phase shift method, and finallythe height information of the object surface is obtained by using thegeometric relation.

An example of an imaging mode of the single-pixel sensor and anapplication thereof to the three-dimensional computational imaging and adepth measurement is given below. The specific implementation methodincludes a hardware system description and an algorithm reconstruction,which can be summarized as follows. Anew optical coding mode is designedin advance before the experiment, that is, a dot product of a stripematrix and a two-dimensional imaging coding is obtained, such that a newmatrix is generated as an input of the optical modulator, and the newmatrix is used for encoding and modulating structure-depththree-dimensional information of the scene, and a single-pixel detectoris used for performing a coupling and an acquisition to obtain aone-dimensional light intensity superposition value. The reconstructionalgorithm for resolving a single-pixel may perform the two-dimensionalimaging by using the following various methods, but is not limited tothe following methods, and further perform the three-dimensional imagingby using a phase shift method. The method includes the followings.

1. Hardware System Implementation

To build the hardware system, a single-pixel imaging uses a lightmodulator to modulate a light pattern. The reflected or transmittedlight from a target scene is ultimately collected by the single-pixeldetector, as illustrated in FIG. 2. A two-dimensional object scene maybe reconstructed by using various algorithms, such as a non-iterativemethod, a linear iterative method, a non-linear iterative method, and adeep learning method (specifically, as described above).

FIG. 2 shows light paths of two common single-pixel imaging systems: (a)the left drawing shows a light path of an active single-pixel imagingsystem, in the active single-pixel detection system, the light modulatoris located between an active light source and the target scene; (b) theright drawing shows a light path of a passive single-pixel imagingsystem, in the passive single-pixel detection system, the lightmodulator is located between the target scene and a detection module,and this configuration does not require an active light source.

For the hardware system, these two building modes of the active lightsystem and the passive light system are both suitable for the presentdisclosure. The active one is taken as an example for the description,of which an improvement and innovation are shown in FIG. 3. It can beseen that, a pre-coding setting part of the spatial light modulator isprocessed to make a stripe pattern is hidden on a mask image. Thismodulation mode may effectively modulate the three-dimensionalinformation of the scene without adding other elements such as a gratingand the like, and finally enables the detector to receive aone-dimensional measurement value including the three-dimensionalinformation.

2. Algorithm Reconstruction-Two-Dimensional Information Decoupling

A two-dimensional computational imaging scheme based on the single-pixelsensor is a linear system. In particular, a measurement model may bedescribed as:Ax=b,

where A∈R^(m×n) represents a light modulation matrix (m represents anumber of modulation modes, each modulation mode includes n pixels),x∈R^(n×1) represents the target scene to be reconstructed (aligned as avector), and b∈R^(m×1) represents a measurement vector.

The reconstruction of the single-pixel detector refers to calculatingfrom the modulation mode A and the corresponding measurement value b toobtain x. In the present disclosure, a two-dimensional graph may berecovered according to different algorithms, and then the heightinformation of the object surface is solved via the two-dimensionalgraph combined with a geometric relation.

Algorithms for single-pixel two-dimensional imaging may include: thenon-iterative method, the linear iterative method and the non-lineariterative method. The above algorithms include the matrix inversion, theconventional correlated imaging, the differential ghost imaging (DGI),the gradient descent (GD), the conjugate gradient descent (CGD), thePoisson maximum likelihood method, the alternative projection (AP), thesparse representation compressed sensing method (SR-CS), the totalvariation compressed sensing method (TV-CS), etc. Any one of the abovealgorithms may be used to solve a required result according to theimaging requirements of the user.

3. Algorithm Reconstruction-Depth Information Decoupling:

For the three-dimensional reconstruction, a specific principle is to usea phase shift profile method to perform the reconstruction. The methodis based on a sinusoidal grating projection, a deformed light fieldimage is obtained by a method of moving a grating by hardware orsoftware, the phase distribution is calculated according to a N-stepphase shift algorithm according to the discrete phase shift technology,and finally the height information of the object surface is obtained byusing the geometric relation. An optical path structure of the phaseshift method is illustrated in FIG. 4. Taking a point H on the surfaceof the object to be measured in FIG. 4 as an example, the formula may berepresented by:

$h = \frac{{Lp}_{0}{\Delta\phi}}{{2\pi d} + {p_{0}{\Delta\phi}}}$

where, P₀ represents a period of a sinusoidal grating. When both aprojection system and the imaging system are far from the object, afocus bias in the depth range of the object may be negligible. When asinusoidal pattern is projected onto a three-dimensional diffuse object,its distorted light field detected by imaging may be described as:I(x,y)=R(x,y){A(x,y)+B(x,y)cos [ϕ(x,y)]},

where R(x, y) represents a reflectivity of each point on the surface,A(x, y) represents a background light intensity, B(x, y)/A(x, y)represents a stripe contrast, a phase function ϕ(x, y) represents afeature of a stripe pattern, so that the height information of theobject surface shape is included. Since the reflectivity R(x, y) of eachpoint of the object is little changed, it may be assumed as a constantvalue (1 in general), the above formula may be rewritten as follows:I(x,y)=A(x,y)+B(x,y)cos [ϕ(x,y)].

The detection array samples the object, measures a phase of eachsampling point by using the phase shift technology, and records N(N≥3)light intensity values. For each image, the grating moves p₀/N, assumingI₁, I₂, . . . , I_(n) are the light intensity values of a same point C,then

${{\tan\;\phi_{C}} = \frac{\sum\limits_{n = 1}^{N}{{I_{n}( {x,y} )} \times {\sin( {2\pi\;{n/N}} )}}}{\sum\limits_{n = 1}^{N}{{I_{n}( {x,y} )} \times {\cos( {2\pi\;{n/N}} )}}}},$

in which, since the phase determined by the above equation is π-modulo,signs of sin ϕ and cos ϕ must be determined if the determined phase is2π-modulo, the process also known as a phase ambiguity removing.

In order to calculate a height distribution of the object to be measuredfrom the phase function, a wrapped phase obtained by an inversetrigonometric operation should be unwrapped into an original phasedistribution to obtain a continuously distributed two-dimensional phasefunction ϕ_(n)(x, y). For this purpose, a wrapped phase data matrix isunfolded along a row or column direction of the wrapped phase datamatrix, phase values of two adjacent points are compared in theunfolding direction, and if a difference value is smaller than −π, thephase value of the latter point should be added by 2π; if the differenceis greater than π, the phase of the latter point should be subtracted by2π. In the above process, it has been assumed in practice that theunwrapped phase change between any two adjacent samples is smaller thanπ, that is, requirements of a sampling theorem must be satisfied, andthat there are at least two samples per stripe, that is, a samplingfrequency is greater than twice a highest spatial frequency. Theunwrapped phase value of any point on the imaging surface may beutilized to calculate the height value of a corresponding point on theobject surface.

Finally, a software complete algorithm flow chart from a one-dimensionalsingle-pixel measurement to a two-dimensional recovery and athree-dimensional reconstruction completion is given, as illustrated inFIG. 5. Taking an example of recovering a three-dimensional surface of asemicircle, a pixel size is 64×64, a center of the semicircle is (32,32), a radius is 18 pixels, and a number of acquired one-dimensionalmeasurement values is 4000, as illustrated in FIG. 5.

In summary, the conventional camera uses a two-dimensional sensor unitarray as a photosensitive element, and the sensor can only sense theintensity superposition information of light, and the depth informationof the scene is lost in the imaging process. In addition, thesignal-to-noise ratio of the traditional camera sensor is low, and asensed spectral range is limited. In order to solve the above problems,embodiments of the present disclosure provide the three-dimensionalcomputational imaging method and apparatus based on the single-pixelsensor. Firstly, the new optical coding mode is designed by combiningthe stripe coding and the two-dimensional imaging coding, so thattwo-dimensional space information and the depth information of the sceneare encoded simultaneously. And then, the coding information is coupledand acquired by using the single-pixel sensor to obtain theone-dimensional measurement data. Finally, the high-precisionreconstruction is performed on the three-dimensional information(including the two-dimensional spatial information and the depthinformation) of the scene by using the decoupling algorithm based on thecompressed sensing, the alternative projection or the depth learningtheory. According to the present disclosure, the depth information ofthe scene may be effectively acquired in the single-pixel imaging systemby changing the optical coding mode and the corresponding decouplingalgorithm without adding any hardware, it has important significance forrecovering an object depth by using a small number of acquisition valuesand simplifying system construction, and has wide application in thefields of three-dimensional imaging, multimedia processing and the like.

According to the three-dimensional computational imaging method based onthe single-pixel sensor, the new optical coding mode is adopted tocombine the stripe coding and the two-dimensional imaging coding, thenthe single-pixel detector is utilized to couple and acquire thethree-dimensional information of the scene, and finally the decouplingalgorithm is used to reconstruct the three-dimensional information ofthe scene; the optical coding mode of combining the stripe coding andthe two-dimensional imaging coding may be realized without changing theoriginal integral optical path system; the decoupling algorithm foranalyzing the three-dimensional information from the one-dimensionalmeasurement value may be realized on hardware systems such as a commoncomputer or a development board, and the like. A simple systemconstruction may be realized, so that the method is convenient for amulti-domain application.

The three-dimensional computational imaging apparatus based on thesingle-pixel sensor according to an embodiment of the present disclosurewill be described next with reference to the accompanying drawings.

FIG. 6 is a schematic structural diagram of the three-dimensionalcomputational imaging apparatus based on the single-pixel sensoraccording to an embodiment of the present disclosure.

As illustrated in FIG. 6, the three-dimensional computational imagingapparatus 10 based on the single-pixel sensor includes a generationmodule 100, a coupling module 200 and a reconstructing module 300.

The generating module 100 is configured to combine a stripe coding witha two-dimensional imaging coding through a preset optical coding,generate a new optical coding, and load the new optical coding into anoptical modulator. The coupling module 200 is configured to couple atwo-dimensional spatial information and depth information of a sceneinto a one-dimensional measurement value by using a single-pixeldetector and the optical modulator loaded with the new optical coding.The reconstructing module 300 is configured to reconstruct, from theone-dimensional measurement value through a decoupling algorithm, thetwo-dimensional spatial information and the depth information of thescene for three-dimensional imaging. The apparatus 10 of embodiments ofthe present disclosure may be applied in a case of a wide spectralrange, may effectively recover a depth of a scene or a thickness of eachpoint of a three-dimensional object to be measured, and solves a problemthat the depth cannot be effectively acquired in a traditionalsingle-pixel imaging through a designed new coding mechanism.

Further, in one embodiment of the present disclosure, the generatingmodule 100 is further configured to preset the matrix of the stripecoding multiplied by the matrix of the two-dimensional imaging coding toobtain the new matrix as the input of the optical modulator, so as toeffectively encode the two-dimensional spatial information and the depthinformation of the scene simultaneously.

Further, in one embodiment of the present disclosure, thetwo-dimensional imaging code includes, but is not limited to, one ormore of the random coding, the Hadamard coding, and the stripe coding.

Further, in one embodiment of the present disclosure, the decouplingalgorithm includes, but is not limited to, one or more of thenon-iterative method, the linear iterative method, the non-lineariterative method the a deep learning method, and the decouplingalgorithm is used to decouple the two-dimensional spatial information,the non-iterative method includes the matrix inversion, the correlatedimaging, and the differential ghost imaging, the linear iterative methodincludes the gradient descent, the conjugate gradient descent, thePoisson maximum likelihood method, and the alternative projectionmethod, and the non-linear iterative method includes the sparserepresentation compressed sensing method and the total variationcompressed sensing method.

Further, in one embodiment of the present disclosure, the reconstructingmodule 300 is further configured to perform the reconstruction throughthe decoupling algorithm to obtain the deformed optical field image.

Further, in one embodiment of the present disclosure, the reconstructingmodule 300 is further configured to calculate the phase distributionthrough the phase measurement profile method, and obtain the heightinformation of the object surface by using the geometric relationaccording to the phase distribution.

It should be noted that the foregoing explanation of embodiments of thethree-dimensional computational imaging method based on the single-pixelsensor is also applicable to the three-dimensional computational imagingapparatus based on the single-pixel sensor of this embodiment, which isnot elaborated here.

According to the three-dimensional computational imaging apparatus basedon the single-pixel sensor, the new optical coding mode is adopted tocombine the stripe coding and the two-dimensional imaging coding, thenthe single-pixel detector is utilized to couple and acquire thethree-dimensional information of the scene, and finally the decouplingalgorithm is used to reconstruct the three-dimensional information ofthe scene; the optical coding mode of combining the stripe coding andthe two-dimensional imaging coding may be realized without changing theoriginal integral optical path system; the decoupling algorithm foranalyzing the three-dimensional information from the one-dimensionalmeasurement value may be realized on hardware systems such as the commoncomputer or the development board, and the like. A simple systemconstruction may be realized, so that the apparatus is convenient forthe multi-domain application.

Embodiments of the present disclosure provide a non-transitorycomputer-readable storage medium having stored therein instructionsthat, when executed by a processor of a device, cause the processor toperform the three-dimensional computational imaging method based on asingle-pixel sensor according to embodiments of the present disclosure.

In addition, terms such as “first” and “second” are used herein forpurposes of description and are not intended to indicate or implyrelative importance or significance. Thus, the feature defined with“first” and “second” may comprise one or more this feature. In thedescription of the present disclosure, “a plurality of” means at leasttwo, for example, two or three, unless specified otherwise.

Reference throughout this specification to “an embodiment,” “someembodiments,” “an example,” “a specific example,” or “some examples”means that a particular feature, structure, material, or characteristicdescribed in connection with the embodiment or example is included in atleast one embodiment or example of the present disclosure. Theappearances of the above phrases in various places throughout thisspecification are not necessarily referring to the same embodiment orexample of the present disclosure. Furthermore, the particular features,structures, materials, or characteristics may be combined in anysuitable manner in one or more embodiments or examples. In addition,different embodiments or examples and features of different embodimentsor examples described in the specification may be combined by thoseskilled in the art without mutual contradiction.

The various device components, modules, units, circuits, sub-circuits,blocks, or portions may have modular configurations, or are composed ofdiscrete components, but nonetheless can be referred to as “modules” ingeneral. In other words, the “components,” “modules,” “units,”“circuits,” “sub-circuits,” “blocks,” or “portions” referred to hereinmay or may not be in modular forms, and these phrases may beinterchangeably used.

In the present disclosure, the terms “installed,” “connected,”“coupled,” “fixed” and the like shall be understood broadly, and can beeither a fixed connection or a detachable connection, or integrated,unless otherwise explicitly defined. These terms can refer to mechanicalor electrical connections, or both. Such connections can be directconnections or indirect connections through an intermediate medium.These terms can also refer to the internal connections or theinteractions between elements. The specific meanings of the above termsin the present disclosure can be understood by those of ordinary skillin the art on a case-by-case basis.

In the description of the present disclosure, the terms “oneembodiment,” “some embodiments,” “example,” “specific example,” or “someexamples,” and the like can indicate a specific feature described inconnection with the embodiment or example, a structure, a material orfeature included in at least one embodiment or example. In the presentdisclosure, the schematic representation of the above terms is notnecessarily directed to the same embodiment or example.

Moreover, the particular features, structures, materials, orcharacteristics described can be combined in a suitable manner in anyone or more embodiments or examples. In addition, various embodiments orexamples described in the specification, as well as features of variousembodiments or examples, can be combined and reorganized.

In some embodiments, the control and/or interface software or app can beprovided in a form of a non-transitory computer-readable storage mediumhaving instructions stored thereon is further provided. For example, thenon-transitory computer-readable storage medium can be a ROM, a CD-ROM,a magnetic tape, a floppy disk, optical data storage equipment, a flashdrive such as a USB drive or an SD card, and the like.

Implementations of the subject matter and the operations described inthis disclosure can be implemented in digital electronic circuitry, orin computer software, firmware, or hardware, including the structuresdisclosed herein and their structural equivalents, or in combinations ofone or more of them. Implementations of the subject matter described inthis disclosure can be implemented as one or more computer programs,i.e., one or more portions of computer program instructions, encoded onone or more computer storage medium for execution by, or to control theoperation of, data processing apparatus.

Alternatively, or in addition, the program instructions can be encodedon an artificially-generated propagated signal, e.g., amachine-generated electrical, optical, or electromagnetic signal, whichis generated to encode information for transmission to suitable receiverapparatus for execution by a data processing apparatus. A computerstorage medium can be, or be included in, a computer-readable storagedevice, a computer-readable storage substrate, a random or serial accessmemory array or device, or a combination of one or more of them.

Moreover, while a computer storage medium is not a propagated signal, acomputer storage medium can be a source or destination of computerprogram instructions encoded in an artificially-generated propagatedsignal. The computer storage medium can also be, or be included in, oneor more separate components or media (e.g., multiple CDs, disks, drives,or other storage devices). Accordingly, the computer storage medium canbe tangible.

The operations described in this disclosure can be implemented asoperations performed by a data processing apparatus on data stored onone or more computer-readable storage devices or received from othersources.

The devices in this disclosure can include special purpose logiccircuitry, e.g., an FPGA (field-programmable gate array), or an ASIC(application-specific integrated circuit). The device can also include,in addition to hardware, code that creates an execution environment forthe computer program in question, e.g., code that constitutes processorfirmware, a protocol stack, a database management system, an operatingsystem, a cross-platform runtime environment, a virtual machine, or acombination of one or more of them. The devices and executionenvironment can realize various different computing modelinfrastructures, such as web services, distributed computing, and gridcomputing infrastructures.

A computer program (also known as a program, software, softwareapplication, app, script, or code) can be written in any form ofprogramming language, including compiled or interpreted languages,declarative or procedural languages, and it can be deployed in any form,including as a stand-alone program or as a portion, component,subroutine, object, or other portion suitable for use in a computingenvironment. A computer program can, but need not, correspond to a filein a file system. A program can be stored in a portion of a file thatholds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more portions, sub-programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this disclosure can beperformed by one or more programmable processors executing one or morecomputer programs to perform actions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA, or an ASIC.

Processors or processing circuits suitable for the execution of acomputer program include, by way of example, both general and specialpurpose microprocessors, and any one or more processors of any kind ofdigital computer. Generally, a processor will receive instructions anddata from a read-only memory, or a random-access memory, or both.Elements of a computer can include a processor configured to performactions in accordance with instructions and one or more memory devicesfor storing instructions and data.

Generally, a computer will also include, or be operatively coupled toreceive data from or transfer data to, or both, one or more mass storagedevices for storing data, e.g., magnetic, magneto-optical disks, oroptical disks. However, a computer need not have such devices. Moreover,a computer can be embedded in another device, e.g., a mobile telephone,a personal digital assistant (PDA), a mobile audio or video player, agame console, a Global Positioning System (GPS) receiver, or a portablestorage device (e.g., a universal serial bus (USB) flash drive), to namejust a few.

Devices suitable for storing computer program instructions and datainclude all forms of non-volatile memory, media and memory devices,including by way of example semiconductor memory devices, e.g., EPROM,EEPROM, and flash memory devices; magnetic disks, e.g., internal harddisks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROMdisks. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

To provide for interaction with a user, implementations of the subjectmatter described in this specification can be implemented with acomputer and/or a display device, e.g., a VR/AR device, a head-mountdisplay (HMD) device, a head-up display (HUD) device, smart eyewear(e.g., glasses), a CRT (cathode-ray tube), LCD (liquid-crystal display),OLED (organic light emitting diode), or any other monitor for displayinginformation to the user and a keyboard, a pointing device, e.g., amouse, trackball, etc., or a touch screen, touch pad, etc., by which theuser can provide input to the computer. Implementations of the subjectmatter described in this specification can be implemented in a computingsystem that includes a back-end component, e.g., as a data server, orthat includes a middleware component, e.g., an application server, orthat includes a front-end component, e.g., a client computer having agraphical user interface or a Web browser through which a user caninteract with an implementation of the subject matter described in thisspecification, or any combination of one or more such back-end,middleware, or front-end components.

The components of the system can be interconnected by any form or mediumof digital data communication, e.g., a communication network. Examplesof communication networks include a local area network (“LAN”) and awide area network (“WAN”), an inter-network (e.g., the Internet), andpeer-to-peer networks (e.g., ad hoc peer-to-peer networks).

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of any claims,but rather as descriptions of features specific to particularimplementations. Certain features that are described in thisspecification in the context of separate implementations can also beimplemented in combination in a single implementation. Conversely,various features that are described in the context of a singleimplementation can also be implemented in multiple implementationsseparately or in any suitable subcombination.

Moreover, although features can be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination can be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingcan be advantageous. Moreover, the separation of various systemcomponents in the implementations described above should not beunderstood as requiring such separation in all implementations, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

As such, particular implementations of the subject matter have beendescribed. Other implementations are within the scope of the followingclaims. In some cases, the actions recited in the claims can beperformed in a different order and still achieve desirable results. Inaddition, the processes depicted in the accompanying figures do notnecessarily require the particular order shown, or sequential order, toachieve desirable results. In certain implementations, multitasking orparallel processing can be utilized.

It is intended that the specification and embodiments be considered asexamples only. Other embodiments of the disclosure will be apparent tothose skilled in the art in view of the specification and drawings ofthe present disclosure. That is, although specific embodiments have beendescribed above in detail, the description is merely for purposes ofillustration. It should be appreciated, therefore, that many aspectsdescribed above are not intended as required or essential elementsunless explicitly stated otherwise.

Various modifications of, and equivalent acts corresponding to, thedisclosed aspects of the example embodiments, in addition to thosedescribed above, can be made by a person of ordinary skill in the art,having the benefit of the present disclosure, without departing from thespirit and scope of the disclosure defined in the following claims, thescope of which is to be accorded the broadest interpretation so as toencompass such modifications and equivalent structures. Althoughembodiments of present disclosure have been shown and described above,it should be understood that above embodiments are just explanatory, andcannot be construed to limit the present disclosure, for those skilledin the art, changes, modifications, alternatives, and variations can bemade to the embodiments within the scope of the present disclosure.

What is claimed is:
 1. A three-dimensional computational imaging methodbased on a single-pixel sensor, comprising: combining a stripe code witha two-dimensional imaging code through a preset optical coding,generating a new optical code, and loading the new optical code into anoptical modulator, comprising: presetting a matrix of the stripe codemultiplied by a matrix of the two-dimensional imaging code to obtain anew matrix as an input of the optical modulator, so as to encodetwo-dimensional spatial information and depth information of a scenesimultaneously; coupling the two-dimensional spatial information and thedepth information of the scene into a one-dimensional measurement valueby using a single-pixel detector and the optical modulator loaded withthe new optical code; and reconstructing, from the one-dimensionalmeasurement value through a decoupling algorithm, the two-dimensionalspatial information and the depth information of the scene forthree-dimensional imaging.
 2. The method of claim 1, wherein thetwo-dimensional imaging code comprises one or more of a random code, aHadamard code, and a stripe code.
 3. The method of claim 1, wherein thedecoupling algorithm comprises one or more of a non-iterative method, alinear iterative method, a non-linear iterative method and a deeplearning method, and the decoupling algorithm is used to decouple thetwo-dimensional spatial information, wherein the non-iterative methodcomprises a matrix inversion method, a correlation reconstructionmethod, and a differential ghost imaging method, the linear iterativemethod comprises a gradient descent method, a conjugate gradient descentmethod, a Poisson maximum likelihood method, and an alternativeprojection method, and the non-linear iterative method comprises asparse representation compressed sensing method and a total variationcompressed sensing method.
 4. The method of claim 1, whereinreconstructing, from the one-dimensional measurement value through thedecoupling algorithm, the two-dimensional spatial information and thedepth information of the scene for three-dimensional imaging comprises:performing reconstruction through the decoupling algorithm to obtain adeformed optical field image.
 5. The method of claim 4, whereinreconstructing, from the one-dimensional measurement value through adecoupling algorithm, the two-dimensional spatial information and thedepth information of the scene for three-dimensional imaging comprises:calculating a phase distribution from the deformed optical field imagethrough a phase measurement profile method; and obtaining heightinformation of an object surface by using a geometric relation accordingto the phase distribution.
 6. A three-dimensional computational imagingapparatus based on a single-pixel sensor, comprising: one or moreprocessors; and a memory storing instructions executable by the one ormore processors; wherein the one or more processors are configured to:combine a stripe code with a two-dimensional imaging code through apreset optical coding, generate a new optical code, and load the newoptical code into an optical modulator by presetting a matrix of thestripe code multiplied by a matrix of the two-dimensional imaging codeto obtain a new matrix as an input of the optical modulator, so as toencode two-dimensional spatial information and depth information of ascene simultaneously; couple the two-dimensional spatial information andthe depth information of the scene into a one-dimensional measurementvalue by using a single-pixel detector and the optical modulator loadedwith the new optical code; and reconstruct, from the one-dimensionalmeasurement value through a decoupling algorithm, the two-dimensionalspatial information and the depth information of the scene forthree-dimensional imaging.
 7. The apparatus of claim 6, wherein thetwo-dimensional imaging code comprises one or more of a random code, aHadamard code, and a stripe code.
 8. The apparatus of claim 6, whereinthe decoupling algorithm comprises one or more of a non-iterativemethod, a linear iterative method, a non-linear iterative method and adeep learning method, and the decoupling algorithm is used to decouplethe two-dimensional spatial information, wherein the non-iterativemethod comprises a matrix inversion, a correlated imaging, and adifferential ghost imaging, the linear iterative method comprises agradient descent, a conjugate gradient descent, a Poisson maximumlikelihood method, and an alternative projection method, and thenon-linear iterative method comprises a sparse representation compressedsensing method and a total variation compressed sensing method.
 9. Theapparatus according to claim 6, wherein the one or more processors areconfigured to perform reconstruction through the decoupling algorithm toobtain a deformed optical field image.
 10. The apparatus according toclaim 6, wherein the one or more processors are configured to calculatea phase distribution through a phase measurement profile method; andobtain height information of an object surface by using a geometricrelation according to the phase distribution.
 11. A non-transitorycomputer-readable storage medium having stored therein instructionsthat, when executed by a processor of a device, cause the processor toperform a three-dimensional computational imaging method based on asingle-pixel sensor, and the method comprises: combining a stripe codewith a two-dimensional imaging code through a preset optical coding,generating a new optical code, and loading the new optical code into anoptical modulator, comprising: presetting a matrix of the stripe codemultiplied by a matrix of the two-dimensional imaging code to obtain anew matrix as an input of the optical modulator, so as to encodetwo-dimensional spatial information and depth information of a scenesimultaneously; coupling the two-dimensional spatial information and thedepth information of the scene into a one-dimensional measurement valueby using a single-pixel detector and the optical modulator loaded withthe new optical code; and reconstructing, from the one-dimensionalmeasurement value through a decoupling algorithm, the two-dimensionalspatial information and the depth information of the scene forthree-dimensional imaging.